[IBCDPE-835] Revamps GX Report Uploads #130
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem:
Currently,
GreatExpectationsRunner
uploads the HTML reports generated by evaluating an expectations suite to a unique Synapse folder for each dataset. The file name is the timestamp for when the report was generated. This poses a couple of problems, including the proliferation of files in those folders (1 peragora-data-tools
run) and difficulty in finding reports of interest within the folders.Solution:
Update
agora-data-tools
to instead generate report files named after the expectation suite/dataset all into one folder. We then rely on Synapse's versioning to keep track of new versions by uploading the reports withforceVersion=True
. This should provide a much better interface for individuals looking to examine reports.Notes:
gx_folder
is now a "global" configuration which points to the "Great Expectations Reports" folder for prod and testing. Datasets with existing expectation suites now havegx_enabled
as part of their individual configuration.gx_enabled
is now checked during data processing to determine if GX should be run against a dataset.test_gx.py
andtest_process
are updated as needed.